DOCUMENT RESUME 


ED 350 031 JC 920 482 

AUTHOR Bowyer, Karen A. 

TITLE The Use of Assessment To Determine Instructional 
Effectiveness. 

INSTITUTION Southern Association of Community, Junior, and 
Technical Colleges. 

PUB DATE Oct 90 a) 

NOTE 6p.; For Project Cooperation reports, see ED 322 946, 


ED 321 825, and ED 317 238. 
AVAILABLE FROM Piedmont Technical College, P.O. Drawer 1467, 
Greenwood, SC 29648 ($3). 


PUB TYPE Reports - Descriptive (141) -- Collected Works - 
Serials (022) 

JOURNAL CIT Southern Association of Community, Junior, and 
Technical Colleges (SACJTC) Occasional Paper; v8 n2 
Oct 1990 

EDRS PRICE MFO1/PCO1 Plus Postage. 

DESCRIPTORS Academic Achievement; *College Outcomes Assessment; 


Community Colleges; Educational Assessment; Outcomes 
of Education; Pretests Posttests; Program 
Descriptions; *Program Effectiveness; *Program 
Evaluation; *School Effectiveness; *Self Evaluation 
(Groups); Student Evaluation; Testing; *Two Year 
Colleges 


ABSTRACT 

In 1988, the American College Testing Program, 
together with the National Council of Instructional Administrators 
and the National Council for Student Development, initiated Project 
Cooperation (PC) to examine outcomes assessment strategies and 
measures of institutional effectiveness among two-year colleges. The 
first initiative of PC was a survey of two-year institutions 
nationwide to identify current and future efforts at student outcomes 
assessment. Among the two-year institutions designated by PC to serve 
as demonstration sites to develop student outcomes assessment models 
were Dyersburg State Community College (DSCC), in Tennessee; Midlands 
Technical College (MTC), in South Carolina; Orangeburg-Calhoun 
Technical College (OCTC), in South Carolina; and Technica: College of 
the Lowcountry (TCL), in South Carolina. At DSCC, the model includes 
assessment of the general education proficiency of graduating 
sophomores, of entering students’ basic skills, and of knowledge of 
DSCC programs, as well as 2-year follow-up studies of alumni, surveys 
of graduating sophomores, follow-up surveys of dropouts, and 
community surveys and focus groups. The critical success factors used 
at MTC to evaluate the college's performance include student 
satisfaction and retention; post-education success; community 
economic development; and organizational development. At OCTC, 
effectiveness is measured through student growth in general education 
knowledge from college entry to exit. TCL has developed a model 
assessment, orientation, and advisement program, and an assessment 
strategy which examines the correlation between entry and outcomes 
eoovSndeu. ansicumencs. A Getailed review of the PC survey is 
provided. (PAA) 
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The Use of Assessment 
to Determine Instructional Effectiveness 


There has been an increasing em- 
phasis on accountability and as- 
sessment of the effectiveness of higher 
education institutions during the past 10 
years. The Tennessee Higher Education 
Commission implemented a Per- 
formance Funding Program in 1979 
which required the assessment of stu- 
dent outcomes. The results of this as- 
sessment determined part of the funding 
for a public institution in Tennessee. In 
1979 it began as 2% of an institution's 
appropriation. Now it is 5.45%. During 
the early 1980s the Southern Asso- 
ciation of Colleges and Schoors (SACS) 
led the other accrediting associations in 
establishing criteria related to insti- 
tutional effectiveness. In 1988 the 
Secretary of the U.S. Department of 
Education mandated that outcomes as- 
sessment had to be included in ac- 
creditation standards for all regions that 
wished to continue receiving federal 
funds. The regional accrediting as- 
sociations have been joined by the disci- 
plinary accrediting associations in their 
focus on outcomes assessment. 


The American College Testing Pro- 
gram (ACT) formed a partnership in 
1988 with the National Council of In- 
structional Administrators (NCIA) and 
the National Council for Student Devel- 
opment (NCSD) to answer questions 
about outcomes assessment and insti- 
tutional effectiveness. NCIA and NCSD 
are councils which are affiliated with the 
American Association of Community 
and Junior Colleges (AACJC). This joint 
effort has been named Project Coopera- 
tion. The following members of Southern 
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Association of Community, Junior and 
Technical Colleges (SACJTC) are dem- 
onstration sites of Project Cooperation 
which have been designated to develop 
models for using student outcomes mea- 
sures to assess institutional effective- 
ness: Dyersburg State Community 
College in Tennessee and Midlands Tech- 
nical College, Orangeburg-Calhoun Tech- 
nical College and Technical College of 
the Lowcountry in South Carolina. 


The first major initiative of Project 
Cooperation was to survey the chief 
executive officers of members of AACJC 
in the fall of 1988 to determine the 
current and future status of assessment 
in the two-year colleges across the 
nation. The survey had a 54 percent 
response rate with 675 respondents. 
Those of you who responded will recall 
that the questionnaire had five sections. 
There were questions on three types of 
student outcomes measures in the first 
three sections: (tT) Academic Progress 
and Employment Outcomes, (2) Student 
Learning Skills Outcomes, and (3) Stu- 
dent Satisfaction Outcomes. The use of 
the measures was examined in the fourth 
section and the fifth section contained 
questions on organizational patterns as- 
sociated with specific assessment ac- 
tivities. 


In the next few paragraphs | will high- 
light some of the results of the survey to 
give the reader a sense of the current use 
of sttudent outcomes measures to demon- 
strate institutional effectiveness. 


In the first section on Student Aca- 
demic Progress and Employment Out- 
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comes the six items which received the 
highest ratings as very important mea- 
sures of institutional effectiveness were 
(with 1 being most important, 2 is next, 
etc.): 


1. Employer satisfaction with grad- 
uates 


2. Employer satisfaction with job 
training/skills enhancement 
courses 


3. The percentage of graduates who 
find employment in their major/ 
field of study 


4. The percentage of students who 
complete their intended program or 
degree at your institution 


5. Grade point average of graduates 
who transfer relative to native stu- 
dents at four year institutions 


6. Number of students who graduate 
with an associate degree. 


Sixty-one percent (61%) of the respon- 
dents indicated that Student Academic 
Progress and Employment Outcomes 
were measured and used to assess insti- 
tutional effectiveness in their institutions. 


The two top measures chosen by 
dents as indicators of institutional effec- 
tiveness to be used in the next 3-5 years 
were: 


1. The percentage of students who 
complete their intended program or 
degree at your institution 


2. The percentage of graduates who 
find employment in their major/field 
of study. 


The second section dealt with Student 
Learning Skills assessment. The three 
basic skills receiving the most votes as 
very important measures for assessing 
institutional effectiveness were: writing 
skills, reading skills and arithmetic (compu- 
tational) skills. The process skills with 
the top ratings for importance in mea- 
suring institutional effectiveness were: 
Critical/analytical thinking skills, 
synthesis/integration skills and aes- 
thetic appreciation skills. The percentage 
of use of these process skills outcomes 
to assess institutional effectiveness was 
very low with a range from 6.2 to 16.0%. 
In this section 43% of the respor.dents 
indicated that Student Learning Skills 
were assessed, but they were not used to 
assess institutional effectiveness 
whereas 34.5% indicated that they were 
used to assess the effectiveness of their 
institution. 


The two measures marked as “most 
important” for determining institutional 
effectiveness within the next 3-5 years 
were: 


1. General Education Competency 
(marked by 15.3% of the respon- 
dents) 


2. Field of Study Competency (marked 
by 12.5% of the respondents). 


When asked to rate the level of satis- 
faction at their college on how measures 
of Student Learning Skills (Section 11) 
were used for assessing institutional 
effectiveness the respondents indicated 
a lower than medium score. 


Currently, assessment seems to be 
focused on basic skills rather than pro- 
cess or general competency areas but in 
the future it is anticipated that the mea- 
surement of general education and field 
of study competencies will surpass basic 
skills. 


It was found in Section III that 44% of 
the institutions do not use Student Satis- 
faction Outcome measures to assess 
institutional effectiveness while 55% of 
them do. 


The three Student Satisfaction mea- 
sures which were identified as most 
important for current and future use in 
assessing institutional effectiveness 
were: 


1. Satisfaction with curriculum in 
general 


2. Satisfaction with courses in major 
area of study 


3. Satisfaction with faculty. 
The results of the first three sections of 


this survey confirm what many of us 
might have anticipated. Basic skills are 


most often measured and used to deter- 
mine institutional effectiveness. These 
skills are usually measured at entry and 
sometimes at levels. 


Respondents indicated medium satis- 
faction with how Student Satisfaction 
Outcomes (Section ttl) and Academic 
Progress and Employment Outcomes 
(Section |) are currently used to assess 
institutional effectiveness. There is very 
little assessment being done which can 
be used to determine “value-added.” 


Section IV of the survey was designed 
to determine how institutions use the 
various types of student outcomes infor- 
mation. Respondents were asked to 
choose from twenty-five typical activities 
and functions of an institution to show 
where each type of outcome measure 
was used. The top uses of Academic 
Progress and Employment Outcomes 
were for accreditation studies/reports, 
institutional planning and curriculum 
development. 


In Part B of Section IV the respondents 
indicated that curriculum development 
was the activity most affected by the use 
of student outcomes measures. Aca- 
demic Progress and Employment Out- 
comes were the measures most often 
mentioned in conjunction with the activ- 
ity of curriculum development. The next 
most frequently mentioned activity was 
course placement. Student Learning 
Skills Outcomes were the measures used 
to make course placements. 


When asked to anticipate the activities 
which would be most affected by the use 
of student outcomes measures in the 
next 3-5 years the respondents picked 
curriculum development and institu- 
tional planning as their top choices. 


Section V was designed to determine 
how institutions are organized for mea- 
surement of student outcomes. A person 
or position was designated at 54.5% of 
the institutions to direct or coordinate 
assessment for the purpose of mea- 
suring institutional effectiveness. The 
three most frequently used titles were: 


1. Director/Coordinator of Institutional 
Research (marked by 26.4% of the re- 
spondents) 


2. VP/Director of Planning and Devel- 
opment (marked by 14.6% of the re- 
spondents) 


3. Dean of Instruction (marked by 7.5% 
of the respondents). 


tn over half (52.9%) of the institutions 
this assessment position reports to the 
president. In 74.5% of the institutions the 
president was responsible for initiating 
the creation of the position. The position 
had been defined for less than one year 
in 25.9% of the cases, for three to five 
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years in 17.8% of the institutions and for 
at least ten yeurs in 17.0% of the insti- 
tutions. 


When asked to indicate under what 
circumstances the process of assessing 
institutional effectiveness was initiated, 
the three items most frequently marked 
were: 


1. standard management practice 
(44.3%) 


2. state officials (31.0%) 
3. curricular reform (26.4%). 


The major obstacles to assessing insti- 
tutional effectiveness through student 
outcomes measures were lack of person- 
nel resources (51.8%), lack of financial 
resources (48.4%), lack of adequate data 
base (44.6%) and lack of adequate mea- 
surement instruments (38.8%). Most insti- 
tutions did not think that student un- 
willingness to cooperate in outcomes 
assessment was a problem. However, 
there is the problem of motivating stu- 
dents to do their best in institutions 
which require students to take various 
assessments when there are no conse- 
quences for the student. 


The results of this survey show that 
there is considerable use of student 
outcomes measures to assess insti- 
tutional effectiveness. The following four 
members of SACJTC which are demon- 
stration sites of Project Cooperation have 
developed the models discussed below 
to assess institutional effectiveness. 


1. Dyersburg State Community Col- 
lege (DSCC), Tennessee —A compre- 
hensive analysis of the image of 
DSCC will be used to determine the 
effectiveness of the institution along 
with a broad based assessment pro- 
gram. The assessment program uses 
the following measures: 


— Assessment of the proficiency 
of graduating sophomores ona 
measure of general education 
outcomes using the ACT- 
COMP objective test 


— Assessment of entering student 
basic skills as a part of a manda- 
tory assessment and placement 
program. Students exiting reme- 
dial or developmental classes 
are required to retake a different 
form of the placement exam to 
determine growth 


— Assessment of knowledge of the 
major field of career program 
areas 


— Follow-up of alumni two years 
after graduation using a survey 
developed for statewide use in 
Tennessee 


— Interview and survey of grad- 
uating sophomores 


— Student follow-up surveys to 
determine why students are drop- 
ping out. 


Perceptions of the image of Dyersburg 
State will be gathered from six groups: 
business and industry leaders, general 
population, high school students, high 
school teachers and counselors, alumni 
and DSCC faculty and staff. A telephone 
survey of the general population will be 
conducted to determine their level of 
knowledge and impressions of Dyers- 
burg State Community College. Focus 
groups will be the principal meth- 
odology used with the community. 


Using the structured interview format 
of focus groups with business leaders, 
OSCC gained valuable information. This 
information will benefit DSCC in a 
number of ways, such as: 


— The report was used in refor- 
mulating the institutional 
mission statement 


— The College gained a list of 
attitudes, basic skills, and spe- 
cific skills desired by employers 
which can be used in curriculum 
review 


— College staff gained insight into 
the problems employers are 
having in providing training and 
what kinds of training they 
prefer 


— Employers identified ways in 
which the College could provide 
training which is more useful 
and more accessible 


— Employers identified specific 
courses Dyersburg State could 
offer their employees 


— Employers shared their percep- 
tions of the future of various 
industries and what training 
needs would be relevant 


— The College gained insight into 
how it can support the increas- 
ing internationalization of busi- 

. ness 


— The College found that it had 
strong community support for 
keeping its technical programs 
up to date and keeping them 
available locally. 


2. Midlands Technical College (MTC) 
in South Carolina — College per- 
formance will be measured relative 
to each of the following six critical 
success factors: 


— Accessible, comprehensive pro- 
grams of high quality 


— Student satisfaction and reten- 
tion 


— Post-education satisfaction and 
success 


— Economic development andcom- 
munity involvement 


— Sound, effective resource man- 
agement 


— Dynamic organizational involve- 
ment and development. 


Aset of measurable criteria and indica- 
tors of effectiveness has been developed 
for each critical success factor at Mid- 
lands Technical College. The assess- 
ment plans are designed to: 


— integrate entry assessment on 
the Collegiate Assessment of 
Academic Proficiency (CAAP) 
as closely as possible into the 
instructional program 


— assess all students in entry-level 
mathematics and English classes 
on CAAP regardless of previous 
course work or completion of 
ASSET (almost one-third of the 
two-year student population 
have their entry skills assessed 
using this instrument.) Data on 
all entry-level students would 
then be available while allowing 
identification of first-time fresh- 
men who had also taken ASSET 
in the research analysis 


— conduct post-CAAP assess- 
ment upon completion of 60 
quarter hours of credit and the 
general education core, since 
transfer students comprise 
twenty-five percent (25%) of the 
college population and few trans- 
fer students remain at Midlands 
Technical College through grad- 
uation. x 


3. Orangeburg-Calhoun Technical Col- 
lege in South Carolina —The goal is 
to measure growth in general educa- 
tion knowledge from entry to exit. 
Three components of the CAAP were 
administered to all associate degree 
candidates entering the college in 
fall 1989: Writing Skills, Numerical 
Skills and Reading Comprehension. 
After the CAAP results were received, 
counseling sessions were held to 
interpret the assessment scores and 
to explain that they would be used to 
measure progress. It is planned that 
the CAAP exit assessment will be 
conducted in March 1991 for stu- 
dents graduating in May and in June 
1991 for those completing degrees 
in August. This schedule will provide 
enough time to interpret the results 
to the students. 
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All entering students take the 
ASSET test to determine placement 
into remedial, developmental or 
college level courses. Through the 
use of the ASSET test, the college 
provides a comprehensive advising, 
counseling, orientation and course 
placement program for entering 
students. 


A study is conducted annually to 
determine the extent to which reme- 
dial and developmental courses pre- 
pare students for entry level general 
education courses. 


. Technical College of the Lowcountry 


(TCL) in South Carolina—The pri- 
mary goal is to determine the correla- 
tion between the entry assessment 
instrument (ASSET) and an out- 
comes assessmeni ‘CAAP). TCL has 
developed a model assessment, orien- 
tation and advisement program 
using ASSET and has co-sponsored 
three national conferences on assess- 
ment and institutional effectiveness 
with ACT on Hilton Head Island. 
TCL's work with Project Coopera- 
tion helps the institution as it re- 
sponds to the eighteen areas identi- 
fied by the South Carolina Commis- 
sion on Higher Education to be con- 
sidered in determining institutional 
effectiveness. The eighteen areas 
are: 


— General education 
— Majors or concentrations 


— Performance of professional pro- 
gram graduates on licensing and 
certification exams 


— Reports of program changes 
that have occurred as a result of 
external program evaluation 


— Alumni follow-up studies 


— Entry-level skills necessary for 
college work 


— Success of entering students in 
meeting college admissions pre- 
requisites 


— Remedial and developmental pro- 
grams 


— Achievement of utudents trans- 
ferring from two to four year 
institutions 


— Analysis of undergraduate reten- 
tion and attrition 


— Minority student and faculty ac- 
cess and equity 


— Academic performance of stu- 
dent athletes 


— Assessment of procedures for 
student development 


— Assessment of library usage and 
collection development pro- 
cedures 


— Assessment of administrative 
and financial processes and per- 
formance 


— Assessment of facilities 
— Assessment of public service 
— Assessment of research. 


Project Cooperation at TCL has been 
named the Project for the Assessrrent of 
Collegiate Education (PACE). The criti- 
cal ingredients in this project have been 
the outstanding positive involvement of 
the students in all phases of the project 
and the strong collaboration of the two 
major divisions of the college, Instruction 
and Student Development. These efforts 
were enhanced by a comprehensive 
public relations program. 


The primary use of the assessment 
data collected for PACE will be the 
improvement of instruction to enhance 
student success. Project results will also 
be used in reports to meet institutional 
effectiveness criteria for the State Board 
for Technical and Comprehensive Educa- 
tion, SACS, the South Carolina Commis- 
sion on Higher Education and other 
appropriate community constituencies. 


SUMMARY 


As the four case studies demonstrate, 
there is a need for multiple sources of 
assessment in order to get the full picture 
of institutional effectiveness. Institutions 
need to have concrete examples of using 
assessment results to improve the insti- 
tution. These examples will help us weigh 
the costs and benefits of assessment. 


Those of us in two year colleges have 
been mainly using standardized exams 


and surveys although utilization of focus 
groups, interviews and criterion- 
referenced instruments are broadening 
our list of measures. It is important that 
we focus on the process which produced 
the scores or outcomes rather than focus- 
ing on the scores. We can take a lesson 
from industry where quality is improved 
by improving the design of the product 
and the production process, not by just 
looking at the finished product. In this 
way we can determine what is working 
and have a better idea of how to allocate 
diminishing resources. 
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